1 link tagged with all of: inference optimization + quail
Click any tag below to further narrow down your results
Links
Quail is a new inference engine that runs AI-SQL queries (SQL with LLM function calls) much faster than existing approaches by co-optimizing query planning and GPU inference. It reduces wasted compute from CPU overhead and KV cache mismanagement, cutting costs by 4-5x compared to naive implementations.
- Running AI-SQL queries on vLLM naively is 27x slower than the theoretical optimum because of CPU scheduling overhead and unnecessary KV cache evictions that force reprocessing of tokens.
- Quail jointly plans queries and inference execution to minimize KV cache waste and keep the GPU busy, achieving 360 documents/second on a medical adverse event filtering task that would cost $0.37 on an H100 versus $1.75 with GPT-5 nano.
- The system handles real workloads like filtering 100,000 IMDB reviews through two AI predicates in under 6 minutes for under $0.37, demonstrating practical viability for production AI-SQL pipelines.